Reinforcement Learning Framework Research Articles

Eliminating Primacy Bias in Online Reinforcement Learning by Self-Distillation.

Excessive invalid explorations at the beginning of training lead deep reinforcement learning process to fall into the risk of overfitting, further resulting in spurious decisions, which obstruct agents in the following states and explorations. This phenomenon is termed primacy bias in online reinforcement learning. This work systematically investigates the primacy bias in online reinforcement learning, discussing the reason for primacy bias, while the characteristic of primacy bias is also analyzed. Besides, to learn a policy generalized to the following states and explorations, we develop an online reinforcement learning framework, termed self-distillation reinforcement learning (SDRL), based on knowledge distillation, allowing the agent to transfer the learned knowledge into a randomly initialized policy at regular intervals, and the new policy network is used to replace the original one in the following training. The core idea for this work is distilling knowledge from the trained policy to another policy can filter biases out, generating a more generalized policy in the learning process. Moreover, to avoid the overfitting of the new policy due to excessive distillations, we add an additional loss in the knowledge distillation process, using L2 regularization to improve the generalization, and the self-imitation mechanism is introduced to accelerate the learning on the current experiences. The results of several experiments in DMC and Atari 100k suggest the proposal has the ability to eliminate primacy bias for reinforcement learning methods, and the policy after knowledge distillation can urge agents to get higher scores more quickly.

Safe Autonomous Driving with Latent Dynamics and State-Wise Constraints

Autonomous driving has the potential to revolutionize transportation, but developing safe and reliable systems remains a significant challenge. Reinforcement learning (RL) has emerged as a promising approach for learning optimal control policies in complex driving environments. However, existing RL-based methods often suffer from low sample efficiency and lack explicit safety constraints, leading to unsafe behaviors. In this paper, we propose a novel framework for safe reinforcement learning in autonomous driving that addresses these limitations. Our approach incorporates a latent dynamic model that learns the underlying dynamics of the environment from bird’s-eye view images, enabling efficient learning and reducing the risk of safety violations by generating synthetic data. Furthermore, we introduce state-wise safety constraints through a barrier function, ensuring safety at each state by encoding constraints directly into the learning process. Experimental results in the CARLA simulator demonstrate that our framework significantly outperforms baseline methods in terms of both driving performance and safety. Our work advances the development of safe and efficient autonomous driving systems by leveraging the power of reinforcement learning with explicit safety considerations.

Reinforcement learning for path planning of free-floating space robotic manipulator with collision avoidance and observation noise

This study introduces a novel approach for the path planning of a 6-degree-of-freedom free-floating space robotic manipulator, focusing on collision and obstacle avoidance through reinforcement learning. It addresses the challenges of dynamic coupling between the spacecraft and the robotic manipulator, which significantly affects control and precision in the space environment. An innovative reward function is introduced in the reinforcement learning framework to ensure accurate alignment of the manipulator’s end effector with its target, despite disturbances from the spacecraft and the need for obstacle and collision avoidance. A key feature of this study is the use of quaternions for orientation representation to avoid the singularities associated with conventional Euler angles and enhance the training process’ efficiency. Furthermore, the reward function incorporates joint velocity constraints to refine the path planning for the manipulator joints, enabling efficient obstacle and collision avoidance. Another key feature of this study is the inclusion of observation noise in the training process to enhance the robustness of the agent. Results demonstrate that the proposed reward function enables effective exploration of the action space, leading to high precision in achieving the desired objectives. The study provides a solid theoretical foundation for the application of reinforcement learning in complex free-floating space robotic operations and offers insights for future space missions.

AARF: Autonomous Attack Response Framework for Honeypots to Enhance Interaction Based on Multi-Agent Dynamic Game

Highly interactive honeypots can form reliable connections by responding to attackers to delay and capture intranet attacks. However, current research focuses on modeling the attacker as part of the environment and defining single-step attack actions by simulation to study the interaction of honeypots. It ignores the iterative nature of the attack and defense game, which is inconsistent with the correlative and sequential nature of actions in real attacks. These limitations lead to insufficient interaction of the honeypot response strategies generated by the study, making it difficult to support effective and continuous games with attack behaviors. In this paper, we propose an autonomous attack response framework (named AARF) to enhance interaction based on multi-agent dynamic games. AARF consists of three parts: a virtual honeynet environment, attack agents, and defense agents. Attack agents are modeled to generate multi-step attack chains based on a Hidden Markov Model (HMM) combined with the generic threat framework ATT&CK (Adversarial Tactics, Techniques, and Common Knowledge). The defense agents iteratively interact with the attack behavior chain based on reinforcement learning (RL) to learn to generate honeypot optimal response strategies. Aiming at the sample utilization inefficiency problem of random uniform sampling widely used in RL, we propose the dynamic value label sampling (DVLS) method in the dynamic environment. DVLS can effectively improve the sample utilization during the experience replay phase and thus improve the learning efficiency of honeypot agents under the RL framework. We further couple it with a classic DQN to replace the traditional random uniform sampling method. Based on AARF, we instantiate different functional honeypot models for deception in intranet scenarios. In the simulation environment, honeypots collaboratively respond to multi-step intranet attack chains to defend against these attacks, which demonstrates the effectiveness of AARF. The average cumulative reward of the DQN with DVLS is beyond eight percent, and the convergence speed is improved by five percent compared to a classic DQN.

Collaborative Video Caching in the Edge Network using Deep Reinforcement Learning

With the enormous growth in mobile data traffic over the 5G environment, Adaptive BitRate (ABR) video streaming has become a challenging problem. Recent advances in Mobile Edge Computing (MEC) technology make it feasible to use Base Stations (BSs) intelligently by network caching, popularity-based video streaming, etc. Additional computing resources on the edge node offer an opportunity to reduce network traffic on the backhaul links during peak traffic hours. More recently, it has been found in the literature that collaborative caching strategies between neighbouring BSs (i.e., MEC servers) make it more efficient to reduce backhaul traffic and network congestion and thus improve the viewer experience substantially. In this work, we propose a Reinforcement Learning (RL) based collaborative caching mechanism where the edge servers cooperate to serve the requested content from the end-users. Specifically, this research aims to improve the overall cache hit rate at the MEC, where the edge servers are clustered based on their geographic locations. The said task is modelled as a multi-objective optimization problem and solved using an RL framework. In addition, a novel cache admission and eviction policy is defined by calculating the priority score of video segments in the clustered MEC mesh network.

Age-of-Information based Multi-UAV Trajectories Using Deep Reinforcement Learning

The use of unmanned aerial vehicles (UAVs) as a unified platform for sensing and communication is especially relevant in environments with inadequate infrastructure. In this paper, a multi-UAV system is constructed for dynamic data collection in a resource-constrained environment. The devised approach involves the implementation of an access platform referred to as the Access UAV (A_UAV). This A_UAV orchestrates the data collection process from Inspection-UAVs (I_UAVs), each equipped with a visual sensor, facilitating the relay of collected data to the cloud. Our approach jointly considers the trajectory scheduling of A_UAV and I_UAV to collect data from specific points in a decentralized manner. Specifically, a Deep Reinforcement Learning framework utilizing an actor-critic network is formulated for A_UAV, aiming to generate an equitable access schedule for I_UAVs. Moreover, trajectory scheduling of A_UAV ensures dynamic data collection while minimizing total system energy and the Age of Information (AoI) of data arriving from I_UAVs. The simulation results validate the performance of our proposed approach against several baselines under different parameter settings.

Narration with Graphics: An AI Generated Story Teller

Storytelling has always been a powerful tool to convey information, connect with people, and make an impact. The aim of our project is to generate story by using the concept of AI and Machine learning .AI generated storytelling has apotential to transform a variety of industries, for marketing and education to entertainment and beyond. By utilizing the capabilities of ML and NLP, stories generated by artificial intelligence can be tailored to meet the specific need of each industry like personalized narratives, interactive and dynamic storytelling, multimodal storytelling, cross domain storytelling. The introduction of GEN-AI can lead to both opportunities and potential challenges among different creator communities which required collaboration from both academic and industry. This topic gathers and analyse vast amount of data from various sources including literature, movies, and social media we are using natural language processing which understand and process human language enabling them to communicate with the user effectively. To use the facial recognition technology, we are using DALL-E, OpenCV which uses model like SVM, decision tree, and knn etc. The objective of the research is to investigate the power of AI software and automating the process of story generation. This exploration involves utilizing various AI models, such as natural language generation (NLG) algorithms, deep learning architectures, and reinforcement learning frameworks, to create compelling and coherent stories across different genres and themes. We're showcasing an innovative system that empowers users to effortlessly create short stories accompanied by relevant images with minimal input. Our final project is accessible through a user-friendly web page interface, allowing individuals to craft their narratives seamlessly. Key Words: Storytelling, Artificial intelligence, Natural language processing, Ai system, Narratives, Automatic, DALL -E, OpenCV.

Federated deep reinforcement learning for mobile robot navigation

Navigation of a mobile robot in an unknown environment ensuring the safety of the robot and its surroundings is of utmost importance. Traditional methods, such as pathplanning algorithms, simultaneous localization and mapping, computer vision, and fuzzy techniques, have been employed to address this challenge. However, to achieve better generalization and self-improvement capabilities, reinforcement learning has gained significant attention. The concern of privacy issues in sharing data is also rising in various domains. In this study, a deep reinforcement learning strategy is applied to the mobile robot to move from its initial position to a destination. Specifically, the Deep Q-Learning algorithm has been used for this purpose. This strategy is trained using a federated learning approach to overcome privacy issues and to set a foundation for further analysis of distributed learning. The application scenario considered in this work involves the navigation of a mobile robot to a charging point within a greenhouse environment. The results obtained indicate that both the traditional deep reinforcement learning and federated deep reinforcement learning frameworks are providing 100% success rate. However federated deep reinforcement learning could be a better alternate since it overcomes the privacy issue along with other advantages discussed in this paper.

SFL-TUM: Energy Efficient SFRL method for Large Scale AI Model's Task Offloading in UAV-Assisted MEC Networks

The convergence of mobile edge computing (MEC) network with unmanned aerial vehicles (UAVs) presents an auspicious opportunity to revolutionize wireless communication and facilitate high-speed internet access in remote regions for mobile devices (MDs) as well as large scale artificial intelligence (AI) models. However, the substantial amount of data produced by the UAVs-assisted MEC network necessitates the integration of efficient distributed learning techniques in AI models. In recent times, distributed learning algorithms, including federated reinforcement learning (FRL) and split learning (SL), have been explored for the purpose of learning machine learning (ML) models that are distributed by sharing model parameters, as opposed to large raw data-sets as seen in traditional centralized learning algorithms. To implement the hybrid method, the model is first trained locally on each UAV-assisted MEC network using SL. Subsequently, the model parameters that have been encrypted are sent to a central server for federated averaging. Finally, after the model has been updated, it is distributed to each UAV-assisted MEC network for local fine-tuning. Our simulations indicate that the proposed split and federated reinforcement learning (SFRL) framework yields comparable high-test accuracy performance while consuming less energy compared to extant distributed learning algorithms. Furthermore, the SFRL algorithm efficiently realizes energy-efficient selection between the SL and FRL methods under different distributions. Numerical results shows that the proposed scheme improves the accuracy by 29.31% and reduced the energy consumption by around 67.34% and time delay by about 7.37%. as compared to the existing baseline schemes.

Multi-agent reinforcement learning framework based on information fusion biometric ticketing data in different public transport modes

Multi-agent reinforcement learning framework based on information fusion biometric ticketing data in different public transport modes

Model-based offline reinforcement learning framework for optimizing tunnel boring machine operation

Model-based offline reinforcement learning framework for optimizing tunnel boring machine operation

Multiagent-based deep reinforcement learning framework for multi-asset adaptive trading and portfolio management

The highly dynamic nature of stock markets has motivated researchers to propose various supervised learning models to assist investors to optimize financial performance. Machine learning models have been used to predict price trends, and approaches have been proposed for portfolio management. However, these studies focus on only one kind of financial issue, and the methods proposed exhibit poor generalizability. We address these problems with a multi-agent portfolio adaptive trading framework based on reinforcement learning to create an automated trading system with the best trading strategy that can be achieved by long-short situation judgment and adaptive capital allocation. We use the TD3 algorithm in the multi-agent algorithm to mitigate the overestimation and overfitting exhibited by traditional value functions and improve training stability. Experimental results show that the proposed framework outperforms single-agent reinforcement learning algorithms while achieving more stable returns.

Continuous control of structural vibrations using hybrid deep reinforcement learning policy

An efficient active structural control scheme by utilizing the collective concepts of data-driven and physics-inspired reinforcement learning (RL) approaches is being introduced here. The controller is designed to deliver optimal feedback forces based on the full-state information, wherein the training process involves the use of deep neural networks (NNs) and a specially designed gradient descent-based sequence within the RL framework. This integration of algorithms results in a unique active control policy that accelerates the learning process, thereby demands considerably less computational resources to develop an optimal and stable controller as compared to existing data-driven approaches. Most importantly, the mentioned data-driven and physics-inspired approaches encompass deep deterministic policy gradient (DDPG) and an iterative gradient-based state feedback control (SFSC) algorithms, respectively, to establish the architecture of the hybrid control policy, referred to as hybrid RL-controller. Additional advantage of the hybrid RL-controller is its ability to operate in continuous state–action spaces, which allows the designed controller to address diverse structural control problems. The staggered performance of the hybrid RL-controller, both in continuous and discrete time, is evaluated in three case studies that involve structures in linear and nonlinear regimes. The outcomes include a detailed comparison of the hybrid RL-controller with the individually designed DDPG and SFSC RL control strategies, as well as the uncontrolled scenario. Furthermore, this study thoroughly investigates the real-life implementation concerns of control strategies, such as perturbations in model parameters and input forces, and time delays in the feedback loop. Finally, the results from this study corroborate that the designed RL-controller showcases superior performance and faster execution time in the feedback loop, making it suitable for the vibration control of multidimensional complex structures.

Ship Network Traffic Engineering Based on Reinforcement Learning

This research addresses multiple challenges faced by ship networks, including limited bandwidth, unstable network connections, high latency, and command priority. To solve these problems, we used reinforcement learning-based methods to simulate traffic engineering in ship networks. We focused on three aspects—traffic balance, instruction priority, and complex network structure—to evaluate reinforcement learning performance in these scenarios. Performance: We developed a reinforcement learning framework for ship network traffic engineering that treats the routing policy as the state and the network state as the environment. The agent generates routing changes and uses actions to optimize traffic services. The experimental results show that reinforcement learning optimizes network traffic balance, reasonably arranges instruction priorities, and copes with complex network structures, greatly improving the network’s quality of service (QoS). Through an in-depth analysis of the experimental data, we noticed that network consumption was reduced by 9.1% under reinforcement learning. Reinforcement learning effectively implemented priority routing of high-priority instructions while reducing the occupancy rate of the edge with the highest occupancy rate in the network by 18.53%.

Intelligent Control of Building Vibrations: A Transformer-Based Deep Reinforcement Learning Framework

Intelligent Control of Building Vibrations: A Transformer-Based Deep Reinforcement Learning Framework

VizNav: A Modular Off-Policy Deep Reinforcement Learning Framework for Vision-Based Autonomous UAV Navigation in 3D Dynamic Environments

Unmanned aerial vehicles (UAVs) provide benefits through eco-friendliness, cost-effectiveness, and reduction of human risk. Deep reinforcement learning (DRL) is widely used for autonomous UAV navigation; however, current techniques often oversimplify the environment or impose movement restrictions. Additionally, most vision-based systems lack precise depth perception, while range finders provide a limited environmental overview, and LiDAR is energy-intensive. To address these challenges, this paper proposes VizNav, a modular DRL-based framework for autonomous UAV navigation in dynamic 3D environments without imposing conventional mobility constraints. VizNav incorporates the Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm with Prioritized Experience Replay and Importance Sampling (PER) to improve performance in continuous action spaces and mitigate overestimations. Additionally, VizNav employs depth map images (DMIs) to enhance visual navigation by accurately estimating objects’ depth information, thereby improving obstacle avoidance. Empirical results show that VizNav, by leveraging TD3, improves navigation, and the inclusion of PER and DMI further boosts performance. Furthermore, the deployment of VizNav across various experimental settings confirms its flexibility and adaptability. The framework’s architecture separates the agent’s learning from the training process, facilitating integration with various DRL algorithms, simulation environments, and reward functions. This modularity creates a potential to influence RL simulation in various autonomous navigation systems, including robotics control and autonomous vehicles.

Integrated operations strategies for shared and privately-owned autonomous vehicles: A deep reinforcement learning framework

The shared autonomous vehicle (SAV), combining shared mobility and vehicle automation, will play a pivotal role in urban transportation. Effectively matching SAVs to stochastic customer demand poses a significant challenge, particularly when addressing microscopic multi-AV coordination to prevent conflicts and congestion within a network. To tackle this issue, we propose a novel traffic control scheme called the Integrated Operation Framework for Order Matching and AV Coordination (I-OFOMAC). This scheme introduces a third-party coordinator responsible for integrating online order matching for the SAV platform and real-time AV coordination for the road manager. Our approach utilizes an offline deep reinforcement learning method to train a state value neural network, which facilitates strategic order matching. Additionally, a neural network approximating bisimulation metrics is trained to enhance exploration and accelerate learning. To achieve scalability in network-level AV coordination, we adopt a scheduled transportation scheme. The coordinator solves the I-OFOMAC during each decision epoch, optimizing order matching and AV coordination. The I-OFOMAC aims to balance the goals of minimizing AV travel delay and maximizing SAV service revenue. We conduct extensive simulation studies in the realistic Wangjing network to evaluate the proposed I-OFOMAC scheme. Compared to two-step benchmark frameworks that implement order matching and AV coordination sequentially, the I-OFOMAC achieves an average increase of 13.28% in platform revenue (ranging from 2% to 22.53%) and a 4.48% reduction in average PAV travel delay (ranging from −6.55% to 20.17%).

A method of network attack-defense game and collaborative defense decision-making based on hierarchical multi-agent reinforcement learning

Faced with the challenges of security strategy design brought about by the complexity of attack behavior and the dynamism of network structure, dynamic hierarchical intelligent defense methods have shown their effectiveness. However, in complex network environments, their application requires a higher level of coordination mechanisms. Therefore, this paper proposes a hierarchical multi-agent reinforcement learning network attack and defense game and cooperative defense decision-making method, which autonomously and efficiently completes the formulation of defense strategies and defense behavior responses. Firstly, we construct a Stackelberg hypergame model of cyberspace conflicts, and under the condition of information loss, characterize the multi-layer dynamic defense coordination response mechanism. Secondly, By utilizing a hierarchical multi-agent reinforcement learning method as the driving force for game evolution, we sequentially solve the Nash equilibrium of the game, and form a dynamic autonomous defense strategy. Finally, we construct a hierarchical multi-agent reinforcement learning framework, which decouples the defense decision problem, reduces the dimension of the defense action space and the exploration difficulty of the strategy space, and learns coordinated defense strategies more efficiently. We used the CybORG (Cyber Operations Research Gym) environment for simulation. We compared and analyzed the autonomously generated cyber defense strategies with other related works, verifying the superior coordination performance of our method in defense strategy generation and control.

Learning robust autonomous navigation and locomotion for wheeled-legged robots.

Autonomous wheeled-legged robots have the potential to transform logistics systems, improving operational efficiency and adaptability in urban environments. Navigating urban environments, however, poses unique challenges for robots, necessitating innovative solutions for locomotion and navigation. These challenges include the need for adaptive locomotion across varied terrains and the ability to navigate efficiently around complex dynamic obstacles. This work introduces a fully integrated system comprising adaptive locomotion control, mobility-aware local navigation planning, and large-scale path planning within the city. Using model-free reinforcement learning (RL) techniques and privileged learning, we developed a versatile locomotion controller. This controller achieves efficient and robust locomotion over various rough terrains, facilitated by smooth transitions between walking and driving modes. It is tightly integrated with a learned navigation controller through a hierarchical RL framework, enabling effective navigation through challenging terrain and various obstacles at high speed. Our controllers are integrated into a large-scale urban navigation system and validated by autonomous, kilometer-scale navigation missions conducted in Zurich, Switzerland, and Seville, Spain. These missions demonstrate the system's robustness and adaptability, underscoring the importance of integrated control systems in achieving seamless navigation in complex environments. Our findings support the feasibility of wheeled-legged robots and hierarchical RL for autonomous navigation, with implications for last-mile delivery and beyond.

Reinforcement learning‐based composite suboptimal control for Markov jump singularly perturbed systems with unknown dynamics

In this article, a model‐free parallel reinforcement learning method is proposed to solve the suboptimal control problem for the Markov jump singularly perturbed systems. First, since fast and slow dynamics coexist in Markov jump singularly perturbed systems, it may lead to ill‐conditioned numerical problems during the controller design process. Therefore, the original system can be decomposed into independent subsystems at different time‐scales by employing the reduced order method. In addition, two model‐free parallel reinforcement learning algorithms are designed to obtain the optimal controllers for the fast and slow subsystems, respectively. Moreover, within the framework of reinforcement learning, the controllers of the Markov jump singularly perturbed systems can be acquired without the systems dynamics. Finally, a numerical example is introduced to prove the effectiveness of proposed algorithms.